3D scene graph inference and refinement for vision-as-inverse-graphics

The goal of scene understanding is to interpret images, so as to infer the objects present in a scene, their poses and fine-grained details. This thesis focuses on methods that can provide a much more detailed explanation of the scene than standard bounding-boxes or pixel-level segmentation - we infer the underlying 3D scene given only its projection in the form of a single image. We employ the Vision-as-Inverse-Graphics (VIG) paradigm, which (a) infers the latent variables of a scene such as the objects present and their properties as well as the lighting and the camera, and (b) renders these latent variables to reconstruct the input image. One highly attractive aspect of the VIG approach is that it produces a compact and interpretable representation of the 3D scene in terms of an arbitrary number of objects, called a 'scene graph'. This representation is of a key importance, as it can be useful e.g. if we wish to edit, refine, interpret the scene or interact with it. First, we investigate how the recognition models can be used to infer the scene graph given only a single RGB image. These models are trained using realistic synthetic images and corresponding ground truth scene graphs, obtained from a rich stochastic scene generator. Once the objects have been detected, each object detection is further processed using neural networks to predict the object and global latent variables. This allows computing of object poses and sizes in 3D scene coordinates, given the camera parameters. This inference of the latent variables in the form of a 3D scene graph acts like the encoder of an autoencoder, with graphics rendering as the decoder. One of the major challenges is the problem of placing the detected objects in 3D at a reasonable size and distance with respect to the single camera, the parameters of which are unknown. Previous VIG approaches for multiple objects usually only considered a fixed camera, while we allow for variable camera pose. To infer the camera parameters given the votes cast by the detected objects, we introduce a Probabilistic HoughNets framework for combining probabilistic votes, robustified with an outlier model. Each detection provides one noisy low-dimensional manifold in the Hough space, and by intersecting them probabilistically we reduce the uncertainty on the camera parameters. Given an initialization of a scene graph, its refinement typically involves computationally expensive and inefficient search through the latent space. Since optimization of the 3D scene corresponding to an image is a challenging task even for a few LVs, previous work for multi-object scenes considered only refinement of the geometry, but not the appearance or illumination. To overcome this issue, we develop a framework called 'Learning Direct Optimization' (LiDO) for optimization of the latent variables of a multi-object scene. Instead of minimizing an error metric that compares observed image and the render, this optimization is driven by neural networks that make use of the auto-context in the form of a current scene graph and its render to predict the LV update. Our experiments show that the LiDO method converges rapidly as it does not need to perform a search on the error landscape, produces better solutions than error-based competitors, and is able to handle the mismatch between the data and the fitted scene model. We apply LiDO to a realistic synthetic dataset, and show that the method transfers to work well with real images. The advantages of LiDO mean that it could be a critical component in the development of future vision-as-inverse-graphics systems

Learning Direct Optimization for scene understanding

We develop a Learning Direct Optimization (LiDO) method for the refinement of a latent variable model that describes input image x. Our goal is to explain a single image x with an interpretable 3D computer graphics model having scene graph latent variables z (such as object appearance, camera position). Given a current estimate of z we can render a prediction of the image g(z), which can be compared to the image x. The standard way to proceed is then to measure the error E(x, g(z)) between the two, and use an optimizer to minimize the error. However, it is unknown which error measure E would be most effective for simultaneously addressing issues such as misaligned objects, occlusions, textures, etc. In contrast, the LiDO approach trains a Prediction Network to predict an update directly to correct z, rather than minimizing the error with respect to z. Experiments show that our LiDO method converges rapidly as it does not need to perform a search on the error landscape, produces better solutions than error-based competitors, and is able to handle the mismatch between the data and the fitted scene model. We apply LiDO to a realistic synthetic dataset, and show that the method also transfers to work well with real images

Massive Dimensionality Reduction for the Left Ventricular Mesh

Statistical emulation is a promising approach for the translation of cardio-mechanical modelling into the clinical practice. However, a key challenge is to find a low-dimensional representation of the heart, or, for the specific purpose of diagnosing the risk of heart attacks, the left-ventricle of the heart. We consider the problem of dimensionality reduction of the left ventricular mesh, in which we investigate three classes of techniques: principal component analysis (PCA), deep learning (DL) methods based on auto-encoders, and a parametric model from the cardio-mechanical literature. Our finding is that PCA performs as well as the computationally more expensive DL methods, and both outperform the state-of-the-art parametric model

Massive Dimensionality Reduction for the Left Ventricular Mesh

Statistical emulation is a promising approach for the translation of cardio-mechanical modelling into the clinical practice. However, a key challenge is to find a low-dimensional representation of the heart, or, for the specific purpose of diagnosing the risk of heart attacks, the left-ventricle of the heart. We consider the problem of dimensionality reduction of the left ventricular mesh, in which we investigate three classes of techniques: principal component analysis (PCA), deep learning (DL) methods based on auto-encoders, and a parametric model from the cardio-mechanical literature. Our finding is that PCA performs as well as the computationally more expensive DL methods, and both outperform the state-of-the-art parametric model

Direct Learning Left Ventricular Meshes from CMR Images

Biomechanical studies of the left ventricle (LV) typically rely on a mesh of finite element nodes for a discrete representation of the LV geometry, which is used in an approximate numerical solution of the cardio-mechanical equations based on finite-element discretisation. This is typically done by first manually annotating cardiovascular magnetic resonance (CMR) scans, second creating a preliminary mesh, third manually correcting the mesh to account for motion. The whole process requires specialist knowledge, is time consuming and prone to human error, which prohibits its common adoption in the clinics. We propose to overcome these shortcomings by applying statistical pattern recognition techniques to CMR images. In particular, we train a convolutional neural network (CNN) to predict the LVM via learning its principal component representation directly from CMR scans. As a useful side-product we obtain a low-dimensional representation of the LVM, which is of interest for surrogate models (emulators) of the myocardium constitutive models

Challenges in Representation Learning: A report on three machine learning contests

The ICML 2013 Workshop on Challenges in Representation Learning focused on three challenges: the black box learning challenge, the facial expression recognition challenge, and the multimodal learning challenge. We describe the datasets created for these challenges and summarize the results of the competitions. We provide suggestions for organizers of future challenges and some comments on what kind of knowledge can be gained from machine learning competitions.Comment: 8 pages, 2 figure

AutoML Workshop

Abstract The ChaLearn AutoML Challenge team conducted a large scale evaluation of fully automatic, black-box learning machines for feature-based classification and regression problems. The test bed was composed of 30 data sets from a wide variety of application domains and ranged across different types of complexity. Over six rounds, participants succeeded in delivering AutoML software capable of being trained and tested without human intervention. Although improvements can still be made to close the gap between human-tweaked and AutoML models, this competition contributes to the development of fully automated environments by challenging practitioners to solve problems under specific constraints and sharing their approaches; the platform will remain available for post-challenge submissions at http://codalab.org/AutoML

The Role of Aquaporin 5 (AQP5) in Lung Adenocarcinoma: A Review Article

Aquaporins (AQPs) are selective, transmembrane proteins, which are primarily responsible for the transport of water and small molecules. They have been demonstrated to play a key role in the development and progression of cancer. Lung adenocarcinoma is the most common primary lung cancer diagnosed in patients in Europe and the USA. The research done so far has provided firm evidence that some AQPs can be biomarkers for various diseases. The objective of this review article is to present a potential role of AQP5 in the development of lung adenocarcinoma. Original papers discussing the involvement of AQP5 in carcinogenesis and containing relevant clinical data were identified. In order to analyze the research material in accordance with PRISMA guidelines, a systematic search of the ScienceDirect, Web of Science, and Pubmed databases was conducted. Out of the total number of 199 papers identified, 14 original articles were subject to analysis. This article presents the pathophysiological role of AQP5 in the biology of lung adenocarcinoma as well as its prognostic value. The analysis substantiates the conclusion that the prognostic value of AQP5 in lung cancer requires further research. Another aim of this paper is to disseminate knowledge about AQPs among clinicians

The Role of Aquaporin 5 (AQP5) in Lung Adenocarcinoma: A Review Article

Aquaporins (AQPs) are selective, transmembrane proteins, which are primarily responsible for the transport of water and small molecules. They have been demonstrated to play a key role in the development and progression of cancer. Lung adenocarcinoma is the most common primary lung cancer diagnosed in patients in Europe and the USA. The research done so far has provided firm evidence that some AQPs can be biomarkers for various diseases. The objective of this review article is to present a potential role of AQP5 in the development of lung adenocarcinoma. Original papers discussing the involvement of AQP5 in carcinogenesis and containing relevant clinical data were identified. In order to analyze the research material in accordance with PRISMA guidelines, a systematic search of the ScienceDirect, Web of Science, and Pubmed databases was conducted. Out of the total number of 199 papers identified, 14 original articles were subject to analysis. This article presents the pathophysiological role of AQP5 in the biology of lung adenocarcinoma as well as its prognostic value. The analysis substantiates the conclusion that the prognostic value of AQP5 in lung cancer requires further research. Another aim of this paper is to disseminate knowledge about AQPs among clinicians